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Using the Rasch Rating Scaie modei, data coiiected from 200 ninth grade students 
in North Suiawesi Indonesia were anaiysed. This anaiysis sought to vaiidate 
instruments deveioped to measure the Democratic Ciimate of Civic Education 
Ciassrooms (DCCEC) and Student Engagement in Civic Education Ciassrooms 
(SECEC). Category used, item and person separation reiiabiiity (ISR and PSR), 
item and person separation index (ISI and PSI), item ordering, person and item fit 
were examined in the anaiysis. The resuits indicated the reiiabiiity of items to be 
used in other studies that invoived students from ciassrooms with simiiar ciimates. 
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As a new democratic country, Indonesia needs to improve democratic values among young people 
through its education system. These values could only be nurtured effectively in a context that 
allowed them to be understood, appreciated and applied. As the classroom is the centre of the 
learning process in Indonesia, the classroom needs to be democratic and engaging in order to 
make learning in it meaningful. In order to evaluate both attributes, it is important to have valid 
instruments for measuring them. 

This study sought to develop and validate instruments for measuring the degree of democratic 
climate of the civic education classroom and of students’ classroom engagement. Measures for the 
democratic climate of classrooms and student engagement were developed based on published 
literature and operationalised using a descriptive type Likert scale with four categories (0=never; 
l=rareiy; 2=sometimes; 3=often). Instrument testing involved 200 Grade 9 students (n=200) in 
North Sulawesi, Indonesia. 

Before using both instruments in further studies, it was necessary to examine their reliability and 
validity. For this purpose, responses were analysed using the Rasch Rating Scale model. The 
analysis involved category used, item and person separation reliability (ISR and PSR), item and 
person separation index (ISI and PSI), item ordering, person and item fit. 

LITERATURE REVIEW 

Democratic Climate of Classroom 

Fraser and Walberg (1991) viewed learning environments as social-psychological contexts where 
learning took place (2001). Johnson and McClure (2004) defined the classroom learning 
environment as a social atmosphere in which learning took place that was sometimes called the 
‘educational environment’ or ‘classroom climate’. Kubow and Kinney (2000) argued that 
classroom climate was related to how teaching was conducted in the classroom setting. More 
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specifically, Moos (1979) conceptualised it as a system that had four variables, namely physical 
environment, organisational aspects, teacher characteristics, and pupil characteristics and the 
classroom climate was viewed as the mediator between these variables that operated through 
interactions among class members, teachers and students. This process was influenced by the 
orientation, the quality and the quantity of interactions and intercommunications between the 
classroom members (Allodi, 2002). These in turn affected student satisfaction, self-concept and 
the learning processes that influenced learning outcomes. 

Research into classroom environments has been carried out over many years. Different studies 
have been undertaken to investigate a variety of aspects related to the effects of the classroom 
environment. These studies have ranged from investigating factors influencing learning 
environments to the students’ perceptions of their classrooms, and the relationship between 
students’ perceptions of their classrooms and their learning outcomes. These studies have been 
extended to cover schools and families (Parsons, 2002). In addition, the research workers have 
conducted studies about the effects of the classroom environment on the learning of different 
subject matter in different parts of the world. Aikin (1942) studied the effects of democratic 
processes in the classrooms and the schools in The Eight Year Study in the United States 
(Morgenstern and Keeves, 1997). Kim, Fisher and Fraser (1999) investigated science classroom 
environments; Waldrip and Fisher (2003) investigated the differences between urban and country 
students’ perceptions of their learning environments (Dorman, 2003); and Guthrie and Cox (2001) 
investigated the school and classroom context that would make students want to engage in reading 
longer. 

From learning environment studies, several theories have been developed and used to investigate 
psychological aspects of learning environments. Those theories are Fewin’s (1951) field theory 
(Rosch, 2002), Bandura’s (1986) social cognitive theory (Compeau, Higgins and Huff, 1999) and 
Bronfrenbrenner’s (1977) ecological model of human development, and Watzalawick’s theory of 
human communication (Allodi, 2002; Watzlawick, Beavin and Jackson, 1968). 

Kubow and Kinney (2000) developed eight characteristics for a democratic classroom to foster 
democracy in the classroom. These characteristics were: (a) active participation; (b) avoidance of 
textbook dominated instruction; (c) reflective thinking; (d) student decision-making and problem- 
solving choices; (e) controversial issues; (f) individual responsibilities; (g) recognition of human 
dignity; and (f) relevance. These characteristics would appear to be meaningful and feasible to 
employ because they represented the nature of civic education. The use of these characteristics 
made it possible to identify a classroom that was capable of providing students with open, active, 
and engaging classroom learning experiences. 

Student Engagement 

Research workers in this field have divided engagement in classroom learning into three 
categories, namely behavioural, cognitive, and emotional engagement. Behavioural engagement 
consisted of actions, such as, following the rules, adhering to the classroom norms, and the 
absence of disruptive behaviours, for example, skipping school or getting into trouble, 
participating in classroom learning and academic tasks, persistence, effort, attention, asking 
questions, and participating in school-related activities. Emotional engagement included students’ 
positive and negative affective reactions in the classroom, students’ emotional reactions to the 
school and the teacher, feeling of being important to the school, and valuing success in school- 
related outcomes. Cognitive engagement was conceptualised in terms of a psychological 
investment in learning, a desire to go beyond the requirements of school, and a preference for 
challenge through being strategic or self-regulating (Fredricks, Blumenfeld, Fiedel, and Paris, 
2003; Fredricks, Blumenfeld, and Paris, 2004). 
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Studies have been undertaken to identify the eorrelation between behavioural engagement and 
learning outeomes for elementary and high sehool students (e.g. Alexander et al., 1993; Alexander 
et al., 1997; Alvermann et al., 1987a; Alvermann et al., 1987b; Ames, 1992; Ames and Areher, 
1988; Conchas, 2001; Finn et al., 1991; Finn et al., 1995; Finn and Rock, 1997; Finn and Voelkl, 
1993; Guthrie and Cox, 2001; Guthrie et al., 2001; Guthrie and Wigfield, 2000; Meece, 2003; 
Miller and Meece, 1997; Miller and Meece, 1999; Voelkl et al, 1999; Walberg, 1979). Other 
studies have focused on the correlation between discipline problems, behavioural disengagement 
and achievement across grade levels (e.g. Aikins et al., 2005; Barker and Gump, 1964; Bates et 
al, 2003; Battistich et al, 2000; Chang, 2003; Chen et al, 2005; Connell et al, 1994; Conrad, 
2004; Denham et al, 2003; Finn, 1989; Finn and Pannozzo, 2004; Fredricks and Eccles, 2002; 
Furrer and Skinner, 2003; Kern et al, 2001; Khine and Fisher, 2004; Ladd et al, 1999; Luster et 
al, 2004; Meehan et al, 2003; Snyder et al, 2003; Spira and Lischel, 2005; Valeski and Stipek, 
2001; Wentzel, 1997). The pronounced finding was that behavioural engagement had long-term 
effects on student performance. The students who were found showing engagement and interest in 
their early grade levels were also found to be performing better in their later years (Lredricks et 
al, 2003). 

Brown (1997), Turner and Scott (1995) emphasised that social discourse in learning communities 
was intrinsically motivating. Lurthermore, Wentzel (1991, 1997, 2002, 2003), Urdan and Maehr 
(1995) demonstrated that students’ possession of pro-social goals led to their constructive social 
behaviours in the classroom. Lredricks, Blumenfeld, Lield and Paris (2002) found that there was a 
unique relation of a challenging and a structured work environment involving students’ affect, 
behaviour and cognition. Skinner and Belmont (1993) also found that there was a reciprocal 
relationship between teacher behaviour and student engagement in the classroom. Teacher 
interactions with students predicted student behavioural and emotional engagement in the 
classroom, both directly and through their effects on student perceptions of their interactions with 
teachers. In addition, Kindermann (1993) and Wentzel (2002) argued that there was an association 
between children peer groups and the amount of engagement the children showed in the 
classroom. 

Palinscar (1998) in his analysis of the theory of constructivism claimed that the growing interest 
in social constructivist perspectives was propelled by recent educational reform efforts 
encouraging students to assume a more active role in their learning, to explain their ideas to one 
another, to discuss disagreements, and to cooperate in the solution of complex problems, while 
teachers participated in the design of these contexts and the facilitation of this kind of activity. 
Prom social constructivist perspectives, according to Palinscar, interactions such as those 
achieved through classroom discussions were thought to provide mechanisms for enhancing 
higher-order thinking. In addition, discourse was argued to be the primary symbolic, mediational 
tool for cognitive development. However, to make it an effective context for learning, discourse 
must be communicative. 

Ryan and Patrick (2001) reported that there was some research supporting the argument that the 
social environment of the classroom would be very important for students’ motivation and 
engagement. Students who felt that they related to schools were more likely to have positive 
expectancies for success and appreciation of schools (Roeser, Midgley and Urdan, 1996). In 
addition, compared to traditional formats, classes organised cooperatively would increase efficacy, 
value, and the goal orientation of students (Turner, Meyer, Cox and Logan, 1998). 

Ryan and Patrick (2001) also argued that students would feel more efficacious about their ability 
to learn and to complete activities successfully when they had a greater array of resources on 
which to draw than if they were only working individually. This emphasised the importance of 
collaborative learning (Randolph, 2000; Wentzel, 2002). 
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Greeno (1998) argued that engagement in activities also depended on the interactions of 
individuals with other people. Belenky (1997) studied activities in which women were supported 
in their participation in reflective problem solving and planning about themselves. She reported 
substantial changes in these women becoming more active and confident in their thinking and in 
their claims to understanding and knowing. Clark, Anderson, Kuo and Kim et al. (2003) also 
argued that collaborative discussions could facilitate children to engage in reasoned 
argumentation. 

Regardless of the distinctions made between different types of engagement, research has indicated 
the importance of student engagement in the classroom and school for both academic success and 
discipline. There was also a strong indication that the classroom and school environment played a 
social and psychological role to facilitate learning among students by providing conducive 
contexts. 

Based on these research studies, it is argued to be important to make students engage in learning 
activities in civic education classrooms in order to provide them with opportunities to obtain 
deeper understanding of the civic values transmitted through meaningful classroom experiences in 
order to enable them to implement them critically and responsibly in their social interactions. 

Even though research has shown that discussions in an open climate fostered civic learning of the 
students, research studies have not specifically investigated the effect of the democratic climate of 
classrooms on student civic learning. Furthermore, even though studies have been undertaken into 
the effects of student engagement on science learning, there has been no study conducted to 
investigate the effects of engagement on the civic learning of the students. This information 
becomes even more meaningful when it is linked to civic education programs in Indonesian 
schools that have sometimes been said to be unproductive in supporting democracy and civic 
learning. 

In order to achieve this goal, it is important to develop instruments to measure the Democratic 
Climate of the Civic Education Classroom (DCCEC) and Student Engagement in Civic Education 
Classroom (SECEC) for use in Indonesian schools. 

INSTRUMENT DEVELOPMENT 

Using the literature on the democratic climate of classroom and student engagement (Fredricks et 
al, 2002, 2003, 2004; Kubow and Kinney, 2000, Torney-Purta, Oppenheim and Famen, 1975; 
Torney-Purta, Eehmann, Oswald, and Schulz, 2001; Torney-Purta, 1984) measures were 
developed. Eight characteristics of a democratic classroom proposed by Kubow and Kinney 
(2000) were modified to form seven scales to measure student perceptions of the Democratic 
Climate of Civic Education Classrooms. In constructing items for those seven scales, some items 
that were used in the lEA study for Civic Education in 2001 were adapted with some 
modification. 

The Student Engagement measures were adapted from the scales developed by Fredricks et al 
(2003) that were basically used to assess primary school student engagement. Modification was 
done on this scale to suit the need for measuring classroom engagement in civic education 
classrooms for ninth grade students in North Sulawesi, Indonesia. 

Democratic Climate of Classrooms 

This scale was hypothesised to involve seven dimensions: (a) Active participation; (b) Avoidance 
of textbook dominated instruction; (c) Reflective thinking; (d) Student decision-making and 
problem-solving choices; (e) Controversial issues; (f) Recognition of the human dignity; and (g) 
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Relevance. The instrument consisted of 33 items using a Likert four-point scale {0=never, 
l=rarely, 2= sometimes, 2= often). 

Student Classroom Engagement 

This scale consists of three hypothesised dimensions. 

Behavioural engagement 

This dimension represented participation in classroom learning and academic tasks such as paying 
attention, asking questions, participating in classroom activities, and absence of disruptive actions. 

Emotional engagement 

This dimension represented students’ positive and negative affective reactions in civic education 
classrooms, such as feeling of interest, boredom, happiness, sadness, and anxiety. 

Cognitive engagement 

This dimension represented psychological investment in learning involving self-regulation, or 
being strategic (Fredricks et al., 2003). These three dimensions were measured by an instrument 
consisting of 19 items using a Likert four-point scale (0=never, l=rarely, 2= sometimes, 3= 
often). 

Those items were initially written in English and subsequently translated into Bahasa Indonesia 
because the respondents were Indonesian-speaking students. Unfortunately, no back-translation 
validation was done to check the accuracy of the initial translation of the items from English into 
Bahasa Indonesia. 


INSTRUMENT VALIDATION 

Sample 

The sample consisting of 200 ninth grade students was a convenience sample. They were selected 
from five different schools with a different number of students in four different regencies in North 
Sulawesi, Indonesia. The questionnaire was given to the students when they were approaching 
their final examination to enter Senior High Schools. 

Rating Scale 

In order to test the unidimensionality of the scale, a rating scale model was employed. The rating 
scale analysis is one of the Rasch measurement model procedures that is similar to the partial 
credit model. The difference is that it does not allow item format, categories and scale step values 
to vary across items (Eox and Jones, 1998; Huang and Page, 2002). 

Reliability 

Using the Rasch model, both item separation index (ISR) and person separation reliability (PSR) 
can also be estimated (Wright and Masters, 1982). Person separation reliability is an estimate of 
how well a person can discriminate persons on the measured variable. This represents the 
replicability of person placement across other items measuring the same construct (Bond and Pox, 
2001). Reliability is assessed using the Cronbach alpha coefficient. According to Wright and 
Master, PSR is calculated as 
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where SAJ' is person varianee adjusted for calibration error and SDJ' is the total observed 

variance among persons on the measured variable. The expected value that indicates perfect 
reliability is 1.0 (Fox and Jones, 1998). 

The interpretation of the person separation reliability suffers from problems when the items fail to 
work together to define a single variable that leads to the usage of an alternative index called 
person separation index (PSI). This is calculated as 



where SA^ is the adjusted person standard deviation and SE^ is the average measurement 
error. 

This concept provides the estimate of the sample standard deviation in standard error units 
(Wright and Masters, 1982). The person separation index is not bound by 0 and 1 and a higher 
value is a better index to differentiate persons on the continuum. These indices are useful to 
compare the use of different scales across different classroom situations (Bond and Fox, 2001; 
Fox and Jones, 1998). These ideas also apply in the formation of the item separation reliability 
(ISR) and the item separation index (ISI). 


Category Used 

Originally the instrument was administered using four response categories (0123). It was 
important to find out the best model for the scoring of the rating scale categories before running 
further analyses in order to obtain more reliable results and more meaningful interpretation of the 
data. In addition, it was essential to identify how and to what extent respondents used the rating 
scale categories (Fox and Jones, 1998, Bond and Fox, 2001). 

In order to find out an appropriate category scheme, analyses based on the values of the estimated 
standard deviations, item and person separation reliabilities and item and person indices were 
compared. Appendix A presents the fit indices for all items at the first run of Rasch analysis and 
Appendix B presents the list of items examined. 

There are five alternative scoring schemes with corresponding category meanings. The five 
schemes, as shown in Table 1, including the original one were constructed to test the quality of the 
measures. These five category scoring schemes were tested using Quest software (Adams and 
Khoo, 1993). However, this software did not provide item and person separation indices. As a 
result, each index was calculated manually using both estimates of standard deviation and 
adjusted standard deviation provided by Quest using the following formula 

SA\ = SD^ -MSE. 

where MSE. is the mean square calibration error (Wright and Master, 1982, p. 91). The 
estimates for these different indices are rounded in Table 2. The analysis showed that for the 
Democratic Climate of Civil Education Classrooms (DCCEC) scale, the preferable schemes 
are 0012, 0112, and 0011 with person separation reliability (PSR) and person separation 
index (PSI) (0.76, 3.42), (0.72, 3.37), and (0.70, 2.46) respectively, whereas their item 
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separation reliability (ISR) and item separation index (ISI) are (0.97, 5.43), (0.97, 5.00), and 
(0.98, 6.71) respectively. 


Table 1, Alternative scoring schemes and category meaning 


Scoring Ssdieme 

Cafegor/ meaning 


Foiir-eateffor," model 

0123 

0 = never 


1 = r,?.rely 


2 = someritnes 


3 = often 


Three-cateaorv model 

0012 

0 = never rarely 


1 = -iometimes 


2 = often 

0122 

0 = never 


1 = rarely 


2 = soinerime^/often 

0112 

0 = nearer 


1 = rareh- 'sometime's 


2 = often 


Two-eategoiv model 

0011 

0 = nea'er/rarely 


1 = soinerimej 'often 


Table 2, Comparison of fit indices for alternative scoring schemes of the two classroom 
environment scales 

Democrtstic Cliiniti:? of Civk Educarioii Oa^srooini! 




Sdieine-i 

ais 

IMS SD 

QMS 

QMS SD 

.Ml t 

OM r 

SA 

RMSE 

Acli,.S£i 

RI 

SI 

0123 

1.01 ' 1.01 

0,22’’0.2® 

1.04.’ 1.04 

0.27./0J5 

-0.06'-0.04 

0.14.0.14 

0.67 

0.95 

0.26 

0,46 

0.45:0.90 

0.96'0.75 

2.58 

2.07 

0012 

LOO 0.9S 

0.22/ 0.25 

1.06' 1.06 

0J5'O,47 

-0.5S/-0.15 

0.02 0 .07 

0,76 

0,89 

0.14 

0.26 

O.SO'0.44 

0.97.0.76 

5.43 

3,42 

0122 

1.00' 1.07 

0.17''0.26 

1.13s 1.13 

0.2S'0.49 

O.Sl .'0.34 

0.67,0.32 

O.SO 

0.S4 

0.17 

0.40 

O.SL'0.43 

0.97'0.73 

4.11 

2J0 

0112 

0,99 0.99 

0.2L0.30 

1.01,' 1.01 

024 0.33 

-022 -'-an 

-O.OS.'O.OO 

0,85 

1,01 

0.17 

0.30 

0.9|,‘0.72 

0.97.'D.72 

5.00 

3.37 

0011 

!.00:'0.99 

O.OS 0.1S 

l.Ol,-’ 1.01 

0,25''0.64 

o.CfO--ao4 

0.05' 0,06 

0.94 

0.95 

0.14 

0.46 

l,09,'1.13 

C>,9S.'0.70 

6.11 

2.46 

Studfiir in Civic Education 

iKmsi'Per^oti': 

Schemes 

SIS 

IMS SD 

QMS 

OMS .50 

MI f 

OMr 



RMSE 

Ad] , SD 

RI 

SI 

0123 

l.OL 1.05 

.21.' .44 

1.00.' 1.00 

0.27.''0.3S 

0.10.''0.05 

-0.02.0.03 

0.78 

0.57 

0.14 

022 

0.49.;'0.69 

0.92.'0.®3 

5 57 

2.5P 

0012 

1.00 1.00 

.23.' .32 

1.00' 1.00 

0.32''0.+0 

-^0.25 -0.05 

-0.11 0.04 

0.S9 

0.66 

0.10 

0,26 

0.5S'0.SG 

0.94.'0.83 

S.90 

2.54 

0122 

1.00 ’ .9S 

.12/ .32 

M: .90 

0.23,''0.44 

O.IL'0.09 

-0.34,0.02 

0.9 

0.66 

0.14 

0.26 

0.64,''0.70 

0.90.0.66 

6.43 

2.54 

0112 

1.01 1.05 

.2L .44 

1.00' 1.00 

0.27'0.3S 

O.tO'0.05 

0.09'0.06 

0.95 

0.68 

0.10 

0,30 

0.?3'1.02 

0.93.'0.S3 

9.50 

■y 27 

0011 


.17' .25 

.94. .94 

0.29,'0.5! 

0.03.'0.07 

-0.1 5.' 0.06 

1,06 

0,79 

0.10 

0.35 

0.8B,''0.90 

0.96'0.67 

10.60 

2.26 


IldS— infix .weiTitf square. *=r starisncz Rl— reliability indey 

OM%=-oufit mean B}yllE=-rooi mean square e?ror SI=separancdi index 

SD=s:andafd de\iancn 5*4= standard dexiadon 


For the SECEC scale, the superior category schemes are the same as DCCEC scale. Their PSR 
and PSI are (0.83, 2.54), (0.83, 2.27), and (0.67, 2.26), whereas their ISR and ISI are (0.94, 8.90), 
(0.93, 9.50), and (0.96, 10.60) respectively. 

ISR and ISI for both scales show superior reliability to PSR and PSI for measured variable. This 
indicates that item order was more reliable than person order (Pox and Jones, 1998), that are direct 
consequences of having more persons than items being involved in the measurements made. 

In running the next analysis, model 0011 {never/rarely, sometimes/often) was selected for the 
DCCEC scale and Student Engagement in Civil Education Classrooms (SECEC) scale as well. 
This model showed superiority over the other models. Perhaps the idiosyncratic nature of 
categories affected respondent styles. Overall, the infit mean square (IMS) for this model stayed 
stable after deleting misfitting items both for the DCCEC and SECEC scales from 1.00 (0.13) to 
1.01 (0.09) and from 0.99 (0.17) to 0.98 (0.11) respectively. 
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Item Ordering 

Examining the order of items along the continuum is one of the practices recommended to 
establish the validity of measures (Fox and Jones, 1998). 

Figure 1 shows the ordering of the Democratic Climate items and persons on the calibrated scale. 
The 33 items for DCCEC scale were entered in the analysis using dichotomous scoring 
{00\\=never/rarely, sometimes/often). The threshold is the default representation of item 
difficulty used by Quest. For an item step, this is the ability level that is required for a person to 
have a 50 per cent chance of passing the step. 

Persons Items 

To 


xxxxxxxxxxx 


xxxxxxxxxxxxxxx 


xxxxxx 


xxxxxxxxxxxxxx 


1.0 


0.0 


xxxxxxxxxx 

xxxxxxxxxxxx 

xxxxxxxxxxxxxxxx 

xxxxxxxxxxxxxxxx 

XXXXXJfXXXXXXXXXXXXX 

xxxxxxxxxxxxxxxxxxxxx 

xxxxxxxxxxxxxxxx 


xxxxxxxxxx 


XXXXX3CXXXXXX 


10 


20 26 
22 24 27 

31 

.9 19 52 33 

30 

16 

5 17 

4 


XXXX3CXXXXX 

XX 

XX 


- 3^0 

Each .Y rep7^ese7?Ts ofie person 

Figure 1, Item ordering for Democratic Climate of the Civic Education Classroom 

In general, the ordering of items indicates logical meaning. For instance, the extreme location of 
Item 10, “We do not use a text book for the Citizenship Education class, but we discuss materials 
from newspapers, magazines, and TV shows," is consistent with the real situation of the way 
teaching is conducted in North Sulawesi. Most teachers use the text book as the primary medium 
of instruction. This, to some extent, precludes teachers from introducing flexibility into the 
classroom. Figure 1 shows that Item 10 comes together with Items 23, 20, 26, 22, 24 and 27 as the 
most difficult to agree with. These items involve activities that are less likely to be achieved in the 
class as long as the text book dominates instruction. Items 31, 9, 19, 32, 33 and 30 make up a 
group of items in the middle location. These items are related to the relevance of material taught 
and student decision-making. Items 2, 25, 4, 5 and 17 form the easiest group of items that reflect 
the conventional situation of school classrooms in North Sulawesi. Teachers are usually willing to 
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share ideas with students in order to assess student performanee. Students are discussing political 
or social issues related to the materials taught where sharing ideas among class members occurs. 
Finally students are asked to review their previous lessons at home or in the class. 

Figure 2 shows the ordering of items and persons of the calculated scale for Student Engagement. 
Regarding the ordering of items for the SECEC scale, the scale seems to be ordered in a 
meaningful way. Item 7, “/ lead group discussion in the Citizenship Education classroom, is 
found to be the most difficult item in the scale because not every student can have this opportunity 
in every class. Item 4, “/ follow the rules in the Citizenship Education classroom” turns to be the 
easiest item. This reflects the effects of teacher power in the classroom in North Sulawesi, 
Indonesia on student behaviour. 


Persons 


3.0XXXXXX:CCCXXXXXXXXXXXX?lXXXXX 


Items 


XXXXXXXXXXXXXXXXXXXX 1 

1 

2.0 ! 

I 

i 

! 

XXXXXXXXXXXXXXXXXXXXXXXXX ! 


XXXXXXXXXXXXXXXXX ! 

15 


XXXXXXXXXXXXXXXXX 1 

9 14 

17 

XX3CXXXXXXXXXXXXXX 6 


XXXXXXXXXXXXXXXXXXXXXXXXX j 19 


0.0 


XXXXXXXXX 


XXXX 


13 


3 


XXXX 

XX 

XXXXXXX 


30 

S 12 
4 ii 


X 


- 2.0 


Each A' represents one person 


Figure 2, Item ordering for Student Engagement in the Civic Education Classroom 

Students tend to obey the classroom rules and are reluctant to protest against the established rules. 
Eigure 2 also shows that the grouping pattern of items makes sense. The item estimate produces 
four different groups. These groups are Group 1, Items 7 and 15; Group 2, Items 9, 14, 17, 6 and 
19; Group 3, Items 18, 2, 13 and 3; and Group 4, Items 10, 8, 12, 4 and 11. This item group 
configuration is formed in a way that indicates the reality of student engagement style in the Civic 
Education classroom in North Sulawesi. Eor example, in Group 2, these items are about being 
bored, studying Civic Education materials at home when there is no test, checking Civic 
Education work for mistakes, asking questions in class, and reading extra books about citizenship. 
When those items come into the second level of item difficulty simply because students in North 
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Sulawesi can be said to enjoy Civic Education class, but they do not make much effort to learn the 
subject. 


Person and Item Fit 

In order to examine person and item fit, the Rasch model provides four fit statistics: infit and 
outfit statistics that are expressed as infit mean square or outfit mean square indices (Huang and 
Page, 2002; Wright and Masters, 1982). The infit mean square (IMS) index is preferred to the 
others, because it is found to be less sensitive to the sample size (Fox and Jones, 1998). Infit mean 
square values in the range of 0.78 to 1.30 are used (Bond and Fox, 2001). Values less than 0.78 
generally indicate significant overfit, and values greater than 1.30 indicate significant underfit. 
Both underfitting and overfitting items are generally considered misfitting. 

In assessing items, discrimination and differential item functioning (DIF) are also taken into 
consideration. Item discrimination is the ability of an item to separate respondents into high and 
low ability. Its values correspond to the steepness of curves, and the greater steepness is 
preferable. Differential item functioning (DIF) analysis involves examining how different groups 
of respondents perform on the scale (Adams, 1992; Fayers and Machin, 2002). 

Three items of the 33 DCCEC scale items were underfitting and eleven items have low 
discrimination index (<0.30). These undefitting items are Items II, 12 and 13, whereas low 
discrimination items are Items I, 3, 6, 7, 8, 14, 15, 18, 21, 28 and 29. The discrimination index for 
fitting items is in the range of 0.30 to 0.59. All items of the 19 SECEC scale have fitting MS 
values. However, three of them have low discrimination values. These items are 1,5 and 16. 

With respect to the item DIF, overall values for both scales after trimming are insignificant in 
terms of respondent gender (Table 3). Three items of the DCCEC scale and one item of the 
SECEC scale have significant DIF. These items are 4, 23 and 32 of DCCEC scale and item 
number 18 of the SECEC scale. These items should be used in the future studies with caution. 

In the final analysis of items, ISR and ISI for both scales show superior reliability to the PSR and 
PSI for the measured variable. This indicates that item scale order was more reliable than person 
scale order. ISR for DCCEC is 5.61, and its PSI is 2.63, whereas for SECEC is 5.26, and its PSI is 
1.96. 


CONCLUSION 

The Rasch Rating Scale model was used in the analysis of the data on the Democratic Climate of 
Civic Education Classroom (DCCEC) scale and the Student Engagement in Civic Education 
Classroom (SECEC) scale. The study reveals five important points for both scales. First, the item 
reliabilities for both scales are higher than person reliabilities. This makes the items more reliable 
to use regardless of respondents in other studies. Second, the original category scheme (0123) set 
for the scales proved to be inferior to other models in the analysis. Instead, 0011 model is more 
superior than the others. This reveals that ‘rarely’ and ‘sometimes’ categories are not very 
effective in detecting variation in the responses of the students in the sample. Third, the scale 
order and grouping of items for both scales in the scale difficulty estimate would seem to show 
consistency with the contextual situation of the Civic Education class practices in North Sulawesi, 
Indonesia although this statement has not been substantiated in this study. Fourth, IMS values for 
both scales reveal balance in variation. For DCCEC scale, of 19 fitting items (>0.78 to <1.3), 
nine items are over 1.00, nine others are under 1.00, and one item is 1.00, whereas SECEC, six 
items are over 1.00, and ten others are under 1.00. Fifth, overall DIF for both scales shows 
insignificant differences in performance between males and females on the measured variable. 
This indicates that the subgroups are comparable on the scale, and there is little sign of highly 
significant differential item functioning between the sexes. 
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Table 3, Item fit indices for both DCCEC and SECEC 


Item fit for Democratic Climate of Civic Education Classroom scale 


Indices 

DIF in term of gender 

items 

Thi'esholds 

IMS 

QMS 

INFTt 

OUTFTt 

Disc. 


P 

2 

-1.36 

1.10 

1.11 

0.88 

0.51 

0.30 

1.54 

0.21 

4 

-0.74 

0.96 

0.84 

-0.52 

-0.87 

0.46 

4.70 

0.03 

5 

-0.52 

0.97 

0.94 

-0.41 

-0.29 

0.46 

1.90 

0.17 

9 

Q.10 

1.05 

1.02 

0.78 

0.22 

0.43 

0.17 

0.68 

10 

1.14 

1.17 

1.23 

2.01 

1.58 

0.38 

0.18 

0.67 

16 

-0.36 

0.96 

0.87 

-o.eo 

-0.87 

0.49 

0.07 

0.79 

17 

-0.55 

1.00 

0.96 

0.00 

-0.18 

0.44 

0.28 

0.59 

19 

0.10 

0.95 

0.88 

-0.87 

-0.89 

0,51 

0.52 

0.47 

20 

0.52 

1.13 

1.18 

1.97 

1.44 

0.39 

2.52 

0.11 

22 

0.47 

1.02 

1.00 

0.36 

0.02 

0.47 

0.79 

0.37 

23 

0.69 

0.85 

0.80 

-2.27 

-1.72 

0.59 

4,48 

0.03 

24 

0.42 

0.97 

0.90 

-0.39 

-0.83 

0.50 

1.13 

0.29 

25 

-1.22 

1.09 

1.07 

0.92 

0.38 

0.32 

0.02 

0.88 

26 

0..54 

0.85 

0.77 

-2.34 

-1.94 

0.59 

2.83 

0.09 

27 

0.47 

1.03 

1.00 

0.45 

0.07 

0.46 

0.33 

0.57 

30 

-0.05 

0.90 

0.88 

-1 .68 

-0.89 

0..55 

0.11 

0.74 

31 

0.19 

1.09 

1.16 

1.40 

1.26 

0.41 

0.00 

0.97 

32 

0.10 

1.00 

0.99 

0.00 

-0.03 

0.47 

5.20 

0.02 

33 

0.07 

1.02 

0.95 

0.40 

-0.37 

0.47 

0.36 

0.55 

tSPJPSR 0.93 0.76 ChiSQ: = 27.15 (df =18. p = 0.08) 

tSI/PSI 5.61 2.63 

Item fit for Student Engagement in Civic Edu. Classroom scale 

2 

-0.18 

0.99 

0.95 

-0.04 

-0.20 

0.47 

0.00 

0.99 

3 

-0.37 

0.95 

0.78 

-0.47 

-1.03 

0.50 

2.26 

0.11 

4 

-1.08 

0.94 

0.77 

-0.34 

-0.66 

0.43 

0.45 

0.50 

6 

0.54 

1.01 

0.96 

0.23 

-0.25 

0.50 

0.07 

0.79 

7 

1.46 

1.10 

1.43 

1.37 

2.89 

0.43 

0.64 

0.42 

8 

-1.03 

0.86 

1.26 

-0.94 

0.87 

0.47 

0.81 

0.37 

9 

0.82 

1.29 

1.50 

3.88 

3.41 

0.36 

3.30 

0.07 

10 

-0.78 

0.90 

0.76 

-0.75 

-0.87 

0.47 

0.89 

0.35 

11 

-1.14 

0.83 

0.62 

-1 .05 

-1.21 

0..50 

0.00 

0.96 

12 

-0.98 

0.89 

0.79 

-0.75 

-0.63 

0.47 

1.18 

0.28 

13 

-0.18 

1.02 

0.88 

0.27 

-0.57 

0.46 

0.00 

0.99 

14 

0.79 

0.90 

0.88 

-1.45 

-0.94 

0.59 

1 .05 

0.31 

15 

1.21 

1.09 

1.18 

1.21 

1.42 

0.46 

0.06 

0.45 

17 

0.65 

0.99 

1.10 

-0.10 

0.75 

0.53 

0.12 

0.73 

18 

-0.05 

0.89 

0.82 

-1.21 

-0.96 

0,56 

4.91 

0.03 

19 

0.30 

1.01 

0.93 

0.18 

-0.38 

0.49 

0.31 

0.58 

ISR/PSR 0.95 0.63 ChiSQ = 16.93.15 |df= 15, p = 0.32) 

ISI/PS! 5.26 1.96 
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APPENDIX A 

Fit Indices for all Items at the First Run of Rasch Analysis 
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0,96 

0.56 

1.12 

0.29 

0.44 

0.41 

0.52 

0,39 

0.25 

0.62 



0,96 0.6 

6.63 2.0 


ChiSQ = 17,92 (df =18, p = 0,46) 
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Developing and validating instruments for measuring democratic climate 


APPENDIX B 


List of items examined 


Descriptive information for ses eii scales of the Democratic Climate of Civic Ed. Classroom Scale 


Scale 

Description 

Item 

1 . Active 
paiticipation 

Shaiing ideas and among 
peers and facilitators; 
teachers do not take 
a stand as an authonU' 
on civic subjects; 
teachers shai'e unpressions 
on class activities with 
students. 

1. We feel free to share ideas m class. (LD) 

2. Teachers share ideas with us. 

3. Teachers expect us to take then opinions as 
ultimate decisions on civic education subjects. (R; LD) 

4. We ai'e encoiu'aged to gi\*e feedback to 
oin classmates' opinions. 

5. Teachers encourage us to give feedback to their 
opmions on social and citizenship issues. 

6. Teachers shaie their impressions of Citizensliip 
Education class activities with us. (LD) 

2. Avoidance of 
textbook 
dominated 
mstnictions 

Avoiding textbooks based 
insti'uctions. 

7. Teachers lectine from the textbook and we take notes. (R; LD) 

8. We work on materials from textbooks. (R; LD) 

9. Teachers present Citizensliip Education materials m class 
fr om textbooks supported with many other resources, like 
newspapers, magazmes. radio, etc. 

10. We do not use a textbook for the Citizensliip 
Education class, but we discuss materials from 
newspaper, magazines, and TV shows. 

3. Reflective 
thinking 

Not focusing on fact or 
date memorisation, 
encoui'agement on 
countering different 
opinions and infonnation 
in open classroom, 
reflection on class 
actn^ities. 

1 1 . Teachers place great emphasis on learning facts 

or dates when presenting histoiy* or pohtical events. (R; MS) 

12. Memonsing dates and facts is the best way to get 
a good mai'k fr'om teachers in tlus class. (R; MS) 

13. Teachers requue us to memorise dates or definitions. (R; MS) 

14. We ai'e encouraged to compare our opmions with our peers. (LD) 

15. Teachers present several sides of an issue when 
explaming it m class. (LD) 

16. We ai*e encouraged to make arguments supporting 
our own opimons. 

17. We aie encouraged to review our previous class activities. 

4. Student 
decision- 
makmg and 
problem-soh'ing 
choices 

Freedom for students to 
examine social or political 
issues in cmcs class; 
freedom for students to 
participate and influence 
the decision making 
in citizensliip classes. 

18. Teachers detenmne the class discussion topics. (R; LD) 

19. We ai’e encouraged to make up oiu' own minds about issues. 

20. We are encomaged to choose t>^es of activities in class. 

21. To form a discussion group, we aie encoui'aged to take 
oiu‘ own initiative. (LD) 

22. We ai'e encom'aged to assess our own class activities. 

5. Controversial 
issues 

Teachei's bring contro\ ersial 
issues mto ci\ics class; 
allow students to discuss 
conrro\ ersial issues 
in chics class. 

23. Teachers raise controversial issues in the class. 

24. Teachers encourage us to discuss political or 
social issues that are contiovei'sial. 

25. We discuss controvei*sial pohtical or social issvies in class. 

6- Recognizmg 
human dlgnit^• 

Be open to discussions; 
exlnbit a willingness to 
listen; avoid mu'eflecth'e 
iudginents; questioning 
the views of peers and 
facilitators. 

26. Other students are willing to Usten when one student 
is explammg his her opinions m front of the class. 

27. We ai’e encouraged not to make a judgment on an issue 
or view without good supporting argmnents. 

28. We feel free to disagi'ee openly with our teachers' views. (LD) 

29. Teachers respect our opmions and encoui'age us to 
express them durmg the class. (LD) 

7. Relevance 

Discussing issues 
relating to the past and 
present discussing issues 
relating to local, national 
and international events. 

30. We ai'e encouraged to raise local political or 
social issues to be discussed m the class. 

31. We are encouraged to identify cmTent 
political events to be discussed in the class. 

32. We ai'e encouraged to examine the link between 
local, national, and mtemational ev’ents in dealmg 
with ciu'rent issues. 

33. We ai'e encomaged to identify the characteristics of 
the past and present political events in class. 
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Lists of Items Examined (Cont’d) 


Df^ci iDtn e infoini.-itioii for ser en scales of the Student Ensaeement in Cir ic Ed. C lassiooin Scale 

Sciile 

Scale descrlDtion 

Item 

1 . Behavioural 
eiisasement 

Positive conducts (adherms to classroom 
nouns }, absence of clisniptive 
behaviour such as skipping schools 
and getting in trouble: 
mvolveineat m learning such as 
effort, persistence, concentration, 
attention, asking questions; and 
contribution to class discussions, 
participation m classroom actaities. 

1. I pav attention m the Citizenship Education 
class. (LD) 

2. Ufien I am m the Citizenship Education class. 
I just act as if I am working. (R) 

3. I complete mv Citizenship Education 
homework on time. 

4. I follow the rules m the Citizeuslup Education 
classroom. 

5. I get into trouble in the Citizenslup Education 
classroom. (R; MS) 

6. I ask questions in mv Citizenship Education 
classroom. 

7. I lead group discussions in the Citizenship 
Education classroom. 

2 Emotional 
engagement 

Students' affective reactions m civics 
classroom including interest, boredom, 
liapipmess. sadness, and auxietv'. 

8. I feel happy m the Citizenship Education class. 

9. I feel bored in the Citizensliip Education class. (R) 

10. I feel excited bv the work m the Citizenship 
Education class. 

11. I like being in the Citizenship Education class. 

12. I am interested m the Citizenship Education 
work at classroom. 

3. Cognitive 
engagement 

Smdents' investment in learning 
involving self-regulation, or being 
strategic. 

13. iilieii I read an aiticle about citizenship. I ask 
mvself questions to make sure that I understand. 

14. I studv Citizenship Education materials at home 
even when I donf have a test. 

15. I hw to watch TV shows about things we ai'e 
doing in the Citizenship Education class, 

16. I talk to the people outside the school about 
what I am learning inthe Citizenship 
Education class. (LD) 

17. I cheek my Citizenship Education class work 

for mistakes. 

18. If I don’t understand what a word means when 

I am reading an aiticle about citizenslup matters. I 
do something to figui'e it out, like loofcmg it up in 
a dictionaiy or asking someone. 

19. I read extra books to learn more about tilings 
we ai'e doma intlie Citizenship Education class. 

A'o.fe5.’ tl) MF,' MissfiTriits. imms: {LDj: Low disaimmahon; R= Reversed 


